Obsidian Metadata

type	concept
sources	Source - Demystifying Evals for AI Agents

pass@k and pass^k (Non-Determinism)

Agent behavior varies between runs, so run multiple trials per task.

pass@k: probability of at least one success in k attempts. It rises with k. Use it when one success is enough (e.g., generate several candidates and pick one). In coding you often care about pass@1.
pass^k: probability that all k trials succeed. It falls with k. With a 75% per-trial success rate, pass^3 = 0.75³ ≈ 42%. Use it for customer-facing agents where users expect reliable behavior every time.
At k = 1 they’re equal. By k = 10 they tell opposite stories: pass@k → ~100% and pass^k → ~0%.

Isolate trials: start each from a clean environment. Shared state (leftover files, caches, git history, resource exhaustion) causes correlated failures or inflated scores.
0% pass@100 with a frontier model usually means a broken task or grader, not an incapable agent.
Track confidence intervals on rates (see Judge Bias Correction).

thought umwelt